Image processing method and device

ABSTRACT

The present disclosure relates to an image processing method and a device, wherein the image processing method includes: acquiring a first image, the first image including an image formed with respect to a skin part of a patient to be diagnosed which needs diagnosis; inputting the first image into a neural network, to acquire position information of a pathologic change area in the first image; acquiring a boundary of the pathologic change area in the first image, acquiring an original image and a mask image which include the pathologic change area from the first image; fusing the mask image and the original image, obtaining a target image corresponding to the pathologic change area; wherein the target image is an image for diagnosing the pathologic change area, pixel points in the target image correspond to those in the original image and the mask image one by one.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to China Patent Application No.201710730705.2 filed on Aug. 23, 2017, entitled “Image Processing Methodand Device”, the disclosure of which is incorporated by reference hereinin its entirety.

TECHNICAL FIELD

The present disclosure relates to an image processing method and animage processing device.

BACKGROUND

With respect to a dermatosis patient, the following operations aremainly performed: a suspicious sick area and a surrounding area of thepatient to be diagnosed are photographed, then a practical doctormanually positions the suspicious sick area in the photographed image,and finally, based on a result of the manual positioning, the suspicioussick area is diagnosed.

SUMMARY

According to some embodiments of the present disclosure, there isprovided an image processing method comprising: acquiring a first image,the first image including an image formed with respect to a skin part ofa patient to be diagnosed which needs diagnosis; inputting the firstimage into a neural network, to acquire position information of apathologic change area in the first image; acquiring a boundary of thepathologic change area in the first image, acquiring an original imageand a mask image which include the pathologic change area from the firstimage; and fusing the mask image and the original image to obtain atarget image corresponding to the pathologic change area; wherein thetarget image is an image for diagnosing the pathologic change area,pixel points in the target image correspond to pixel points in theoriginal image and pixel points in the mask image one by one.

In some embodiments, the fusing the mask image and the original image toobtain the target image corresponding to the pathologic change areaincludes: acquiring a first pixel value of each pixel point in the maskimage and a second pixel value of each pixel point in the originalimage; fusing the first pixel value of each pixel point in the maskimage and the second pixel value of a corresponding pixel point in theoriginal image, to acquire a target pixel value of each pixel point; andforming the target image of the pathologic change area according totarget pixel values of all pixel points.

In some embodiments, the fusing the first pixel value of each pixelpoint in the mask image and the second pixel value of the correspondingpixel point in the original image to acquire a target pixel value ofeach pixel point includes: dividing the first pixel value of each pixelpoint in the mask image by a maximum pixel value to obtain a ratio, andmultiplying the ratio by the second pixel value of the correspondingpixel point in the original image, to obtain a first data; acquiring adifference value between the maximum pixel value and the first pixelvalue; adding the first data and the difference value to obtain thetarget pixel value of each pixel point.

In some embodiments, the acquiring the first pixel value of each pixelpoint in the mask image and the second pixel value of each pixel pointin the original image includes: extracting the first pixel value of eachpixel point in the mask image, and using the first pixel value of eachpixel point to constitute a first matrix of the mask image, wherein aposition of the first pixel value of each pixel point in the firstmatrix is determined by a position of the pixel point in the mask image;and extracting the second pixel value of each pixel point in theoriginal image, and using the second pixel value of each pixel point toconstitute a second matrix of the original image, wherein a position ofthe second pixel value of each pixel point in the second matrix isdetermined by a position of the pixel point in the original image thefusing the first pixel value of each pixel point in the mask image andthe second pixel value of the corresponding pixel point in the originalimage to acquire a target pixel value of each pixel point includes:dividing the first matrix by the maximum pixel value, to obtain a thirdmatrix; multiply a pixel value of each pixel point in the third matrixby the second pixel value of a corresponding pixel point in the secondmatrix, to obtain a fourth matrix; subtracting the first pixel value ofeach pixel point in the first matrix from the maximum pixel valuerespectively, to obtain a fifth matrix; and adding the fourth matrix andthe fifth matrix, to obtain a sixth matrix, wherein a pixel value ofeach pixel point in the sixth matrix is the target pixel value of eachpixel point.

In some embodiments, after the fusing the mask image and the originalimage to obtain the target image corresponding to the pathologic changearea, the method further includes: diagnosing the pathologic change areain the target image to acquire a diagnosis result.

In some embodiments, before acquiring the first image, the methodfurther includes: acquiring a sample image; acquiring mark data of thesample image, the mark data including position information of thepathologic change area in the sample image; and by the sample image andthe mark data, training a neural network, to form the neural networkwhich has a required function.

In some embodiments, before by the sample image and the mark data,training the neural network, to form the neural network which has arequired function, the method further includes: randomly selecting aselected proportion of the sample images to be subjected to a hairsupplement process; and/or, randomly selecting a selected proportion ofthe sample images to be subjected to a color enhancement process.

In some embodiments, the position information includes centercoordinates and a radius value of the pathologic change area.

In some embodiments, the position information includes centercoordinates, a major axis radius value and a minor axis radius value ofthe pathologic change area.

In some embodiments, the position information further includes centercoordinates and a radius value of the sample image.

In some embodiments, before inputting the first image into the neuralnetwork to acquire the position information of the pathologic changearea in the first image, the method further includes performingpretreatment on the first image.

In some embodiments, performing pretreatment on the first imageincludes: acquiring a size of the first image; comparing the size of thefirst image with an image resolution parameter of an input layer of theneural network to determine whether the size of the first image is morethan the image resolution parameter of the input layer; in response todetermining that the size of the first image is more than the imageresolution parameter of the input layer, cutting or reducing the firstimage; and in response to determining that the size of the first imageis less than the image resolution parameter of the input layer,enlarging the first image.

In some embodiments, the step of acquiring the boundary of thepathologic change area in the first image to acquire the original imageand the mask image which include the pathologic change area from thefirst image is based on the position information and an image edgedetection algorithm.

In some embodiments, acquiring the first image includes scanning theskin part of the patient to be diagnosed which needs diagnosis to formthe first image.

In some embodiments, fusing the mask image and the original imageincludes performing a bitwise AND operation on the mask image and theoriginal image.

According to some other embodiments of the present disclosure, there isprovided an image processing device comprising: a first acquisitionmodule configured to acquire a first image to be diagnosed, the firstimage including an image formed with respect to a skin part of a patientto be diagnosed which needs diagnosis; a machine learning moduleconfigured to input the first image into a neural network, to acquireposition information of a pathologic change area in the first image; anextraction module configured to acquire a boundary of the pathologicchange area in the first image, acquire an original image and a maskimage which include the pathologic change area from the first image; afuse module configured to fuse the mask image and the original image,obtain a target image corresponding to the pathologic change area;wherein the target image is an image for diagnosing the pathologicchange area, pixel points in the target image correspond to those in theoriginal image and the mask image one by one.

According to some other embodiments of the present disclosure, there isprovided a computer device comprising: a processor; a memory; andcomputer program instructions stored in the memory, which, when executedby the processor, cause the processor to execute one or more steps ofthe image processing method provided by at least one embodiment of thepresent disclosure.

According to some other embodiments of the present disclosure, there isprovided a non-transient computer-readable storage medium with computerprograms stored thereon, which, when executed by a processor, cause theprocessor to execute one or more steps of the image processing methodprovided by at least one embodiment of the present disclosure.

Additional aspects and advantages of the present disclosure will bepartially given in the following description, a part of them will becomeobvious from the following description, or be learned by practices ofthe present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or additional aspects and advantages of the presentdisclosure will become obvious and easily appreciated from the followingdescription of embodiments in conjunction with the drawings, wherein:

FIG. 1 is a flow diagram of an image processing method provided by someembodiments of the present disclosure;

FIG. 2 is a mask image which includes a pathologic change area providedby some embodiments of the present disclosure;

FIG. 3 is an image of one pathologic change area provided by someembodiments of the present disclosure;

FIG. 4 is a flow diagram of another image processing method provided bysome embodiments of the present disclosure;

FIG. 5 is a flow diagram of still another image processing methodprovided by some embodiments of the present disclosure;

FIG. 6 is a schematic view of marks of a sample image provided by someembodiments of the present disclosure;

FIG. 7 is a structural diagram of an image processing device provided bysome embodiments of the present disclosure;

FIG. 8 is a structural diagram of another image processing deviceprovided by some embodiments of the present disclosure;

FIG. 9 is a structural diagram of a computer device provided by someembodiments of the present disclosure.

DETAILED DESCRIPTION

Hereinafter, detail description of embodiments of the present disclosureis given, examples of the embodiments are shown in the drawings, whereinthe same or similar reference numerals denote the same or similarelements or elements that have substantially the same or similarfunctions throughout. Embodiments described below with reference to thedrawings are exemplary, are intended to explain the present disclosure,and can not be construed to limit the present invention.

In an embodiment of the present disclosure, a so-called neural networkshows good application performance in many applications which processimages, such as object recognition, object detection, objectclassification, etc. Since Constitutional Neural Networks (CNNs), e.g.CNNs which include multiple convolutional layers, may detect features ofdifferent regions and dimensions in an image through differentconvolutional layers, a deep learning method developed based on the CNNis used to perform classification and recognition on the image.

CNNs with various structures have been developed. A conventional CNNgenerally consists of an input layer, a convolutional layer, a poolinglayer, a fully connected layer, i.e., INPUT (input layer) -CONV(convolutional layer) -POOL (pooling layer) -FC (fully connected layer).Wherein the convolutional layer performs feature extraction; the poolinglayer performs dimension reduction on an input feature map; the fullyconnected layer is used to connect all the features, and perform output.

As described above, the applicant uses the CNN to describe a basicconcept of an application of the neural network in a field of imageprocessing, this is merely exemplary. In a field of machine learning,neural networks with various structures may be used in applications suchas image processing, etc. Even the CNNs, in addition to the above listedconventional CNNs, may also be a Fully Convolutional Neural Network FCN,a Segmentation Network SegNet, Dilated Convolutions, anatrous-convolution-based deep neural network DeepLab (V1&V2), amulti-scale-convolution-based deep neural network DeepLab (V3), amulti-channel segmentation neural network RefineNet, etc.

Hereinafter, with reference to the drawings, image processing methodsand devices of some embodiments of the present disclosure are described.

FIG. 1 is a flow diagram of an image processing method provided by someembodiments of the present disclosure.

As shown in FIG. 1, the image processing method includes the followingsteps:

S101, acquiring a first image, the first image including an image formedwith respect to a skin part of a patient to be diagnosed which needsdiagnosis.

In an embodiment of the present disclosure, it is possible to form theimage of the skin part by various skin imaging devices, the presentdisclosure does not limit it, each of a skin surface microscope, skinultrasound, a laser confocal scanning microscope, etc, may be applied tothe present disclosure.

For example, it is possible to put, the skin part of the patient to bediagnosed which needs diagnosis, in a photographing area of the skinsurface microscope, photograph it by the skin surface microscope, obtainthe first image for diagnosis.

Wherein, the adopted skin surface microscope may adopt a polarizing skinsurface microscope or a non-polarizing skin surface microscope.

For example, a camera head on a mobile terminal such as a cellphone, atablet, a camera, etc, is used to photograph the skin part which needsdiagnosis, obtain the first image for diagnosis.

S102, inputting the first image into a neural network, to acquireposition information of a pathologic change area in the first image.

In some embodiments, the method further includes a step of performingpretreatment on the first image. For example, a size of the first image,such as length and width of the image, is acquired, the size of thefirst image is compared with an image resolution parameter of an inputlayer of the adopted neural network. When the size of the first image ismore than the image resolution parameter of the input layer, the firstimage is cut or reduced; when the size of the first image is less thanthe image resolution parameter of the input layer, the first image isenlarged.

For example, the image resolution parameter of the input layer (INPUT)of the adopted neural network is 32*32, the resolution of the firstimage is 600*600, the first image may be scaled to the resolution 32*32.

For example, the image resolution parameter of the input layer (INPUT)of the adopted neural network is 32*32, the resolution of the firstimage is 10*10, the first image may be stretched to the resolution32*32.

The neural network processes the first image, outputs positioninformation of the pathologic change area in the first image.

In some embodiments, the position information may include centercoordinates, a radius value, etc, of the pathologic change area.Thereby, according to the center coordinates and the radius value, it ispossible to obtain a circular area which includes the pathologic changearea.

In some embodiments, the position information may include centercoordinates, a major axis radius value, a minor axis radius value, etc,of the pathologic change area. Thereby, according to the centercoordinates, the major axis radius value, the minor axis radius value,it is possible to obtain an elliptical area which includes thepathologic change area.

Those skilled in the art may understand that, other closed geometriesmay also be used to describe the position of the pathologic change area,e.g. they may be a rectangular region or other arbitrarily shaped areaswhich include the pathologic change area.

By the neural network, the circular area or the elliptical area, etc,which includes the pathologic change area, is acquired from the firstimage, this is advantageous to narrowing a range of positioning thepathologic change area. In addition, since medical experts who have richexperience of diagnosing pathological changes of the skin are limited,the neural network may improve the accuracy of the medical staffpositioning the pathologic change area of the skin.

S103, acquiring a boundary of the pathologic change area in the firstimage, acquiring an original image and a mask image which include thepathologic change area from the first image.

In some embodiments, by an image edge detection algorithm, inconjunction with the position information, the original image and themask image which include the pathologic change area are obtained.

For example, according to the center coordinates and the radius value inthe position information, it is possible to obtain the circular areawhich includes the pathologic change area.

For example, according to the image edge detection algorithm, such asSnake algorithm, GAC (Generalized Arc Consistency) algorithm, Level Setalgorithm, etc, the boundary of the pathologic change area is searchedfor from the circular area, and the original image and the mask image ofthe circular area which includes the pathologic change area are acquiredfrom the first image.

For example, FIG. 2 is a mask image which includes the pathologic changearea, wherein a white part in FIG. 2 is a mask image corresponding tothe pathologic change area.

S104, fusing the mask image and the original image, obtaining a targetimage corresponding to the pathologic change area.

After acquiring the original image and the mask image which include thepathologic change area, a fusion process is performed on the mask imageand the original image, to acquire the target image which includes onlythe pathologic change area. The fusion process includes, but not limitedto, performing a bitwise AND operation on the mask image and theoriginal image.

Wherein, the target image includes only the image of the pathologicchange area, does not include an image of a non-pathologic change area,the target image is an image for diagnosing the pathologic change area,pixel points in the target image correspond to those in the originalimage and the mask image one by one. For example, FIG. 3 is the targetimage of the pathologic change area obtained according to the mask imageshown in FIG. 2.

Hereinafter, by one embodiment, how the target image of the pathologicchange area is obtained through pixel values of pixel points in the maskimage and pixel values of pixel points in the original image isintroduced.

As shown in FIG. 4, the image processing method includes the followingsteps:

S401, acquiring a first image, the first image including an image formedwith respect to a skin part of a patient to be diagnosed which needsdiagnosis.

S402, inputting the first image into a neural network, to acquireposition information of a pathologic change area in the first image.

S403, acquiring a boundary of the pathologic change area in the firstimage, acquiring an original image and a mask image which include thepathologic change area from the first image.

Steps S401-S403 are similar to steps S101-S103 in the aforementionedembodiment, repeated description is no longer made herein.

S404, acquiring a first pixel value of each pixel point in the maskimage and a second pixel value of each pixel point in the originalimage.

After acquiring the mask image and the original image, the first pixelvalue of each pixel point in the mask image and the second pixel valueof each pixel point in the original image are respectively acquired.

In some embodiments, the numbers of the pixel points in the mask imageand the original image are the same, and the pixel point positionscorrespond to each other.

S405, fusing the first pixel value of the pixel point in the mask imageand the second pixel value of the corresponding pixel point in theoriginal image, to acquire a target pixel value of the pixel point.

In some embodiments, with respect to each pixel point, the first pixelvalue is divided by a maximum pixel value to obtain a ratio, and theratio is multiplied by the second pixel value of the corresponding pixelpoint in the original image, to obtain a first data. Then, the firstpixel value is subtracted from the maximum pixel value to obtain adifference value. Finally, a value obtained by adding the first data andthe difference value is the target pixel value of the pixel point. Asshown in formula (1).

$\begin{matrix}{d_{pixel} = {{\frac{a_{mask}}{\max}*b_{org}} + \left( {\max - a_{mask}} \right)}} & (1)\end{matrix}$

wherein, d_(pixel) is the target pixel value of the pixel point,a_(mask) is the first pixel value of the pixel point in the mask image,max is the maximum pixel value 255, b_(org) is the second pixel value ofthe corresponding pixel point in the original image. Thus,

$\frac{a_{mask}}{\max}*b_{org}$

is the first data, max−a_(mask) denotes the difference value between themaximum pixel value and the first pixel value.

S406, forming a target image of the pathologic change area according tothe target pixel values of all pixel points.

After acquiring the target pixel value of each pixel point according toformula (1), the target pixel values of all pixel points are put inpositions corresponding to the pixel points in the original image or themask image, the target image which includes only the pathologic changearea may be obtained.

In some embodiments, the pixel value of each pixel point in the maskimage and the pixel value of a corresponding pixel point in the originalimage are fused, to obtain the target pixel value of the pixel point, sothat it is possible to extract the image of the pathologic change area.

In some embodiments, it further includes, before scanning the skin partof the patient to be diagnosed which needs diagnosis to acquire thefirst image for diagnosis, training the neural network to cause it tohave a required function of acquiring the position information of thepathologic change area in the first image. The training method of theneural network, as shown in FIG. 5, may include the following steps:

S501, acquiring a sample image.

In some embodiments, it is possible to acquire an image of a skin partof a patient diagnosed in the past from hospital, as the sample image.

S502, acquiring mark data of the sample image.

After acquiring the sample image, it is possible to mark position dataof a pathologic change area in the sample image by a method of manuallymarking.

For example, taking a circular shape as an example, it is possible tomark center coordinates and a radius value of the pathologic change areain the sample image as well as center coordinates and a radius value ofthe sample image, thereby obtain the mark data of the sample image.Wherein, the mark data includes the center coordinates and the radiusvalue of the pathologic change area in the sample image as well as thecenter coordinates and the radius value of the sample image.

As shown in FIG. 6, by manually marking, center coordinates of a certainsample image is (x1,y1), a radius value is r1, center coordinates of apathologic change area in the sample image are (x2,y2), a radius valueis r2.

By the above mentioned step, it is possible to obtain multiple sampledata sets for training a neural network, each sample in the data setsincludes a sample image as well as marks corresponding to the sampleimage.

S503, by the sample image and the mark data, training a neural network,to form the neural network which has a required function.

The training process of the neural network is widely known in the art.By inputting the sample images and the mark data into the neural networkwith initial parameters, parameters of the neural network arecontinually optimized, to form the neural network which has a functionof acquiring the position information of the pathologic change area inthe first image. With respect to the formed neural network, the input ofthe input layer is an image, the output of the output layer is positioninformation of a pathologic change area.

With respect to the neural network formed by training, the executionprocess is as described in the above embodiment. As shown below, aspecific embodiment is shown.

S504, acquiring a first image, the first image including an image formedwith respect to a skin part of a patient to be diagnosed which needsdiagnosis.

S505, inputting the first image into a neural network, to acquireposition information of a pathologic change area in the first image.

S506, acquiring a boundary of the pathologic change area in the firstimage, acquiring an original image and a mask image which include thepathologic change area from the first image.

Steps S504-S506 are similar to steps S101-S103 in the aforementionedembodiment, therefore repeated description is no longer made herein.

S507, acquiring a first matrix of the mask image and a second matrix ofthe original image.

In some embodiments, a first pixel value of each pixel point in the maskimage is acquired, and the first pixel value of each pixel point isused, to constitute the first matrix of the mask image. Wherein, aposition of the first pixel value of each pixel point in the firstmatrix is determined by a position of the pixel point in the mask image.

As an example, it is possible to use first pixel values of a first rowof pixel points in the mask image as a first row of elements of thefirst matrix, use first pixel values of a second row of pixel points inthe mask image as a second row of elements of the first matrix, that is,the number of rows and the number of columns of the pixel points in themask image are the same as the number of rows and the number of columnsof the first matrix.

As another example, it is possible to use first pixel values of a firstrow of pixel points in the mask image as a first column of elements ofthe first matrix, use first pixel values of a second row of pixel pointsin the mask image as a second column of elements of the first matrix.Therefore, the number of rows and the number of columns of the pixelpoints in the mask image correspond to the number of rows and the numberof columns of the first matrix.

Likewise, a second pixel value of each pixel point in the original imageis extracted, the second pixel value of each pixel point is used, toconstitute the second matrix of the original image. Wherein, a positionof the second pixel value of each pixel point in the second matrix isdetermined by a position of the pixel point in the original image.

S508, obtaining target pixel values of pixel points according to thefirst matrix and the second matrix.

In some embodiments, it is possible to divide the first matrix by amaximum pixel value, that is, divide each first pixel value in the firstmatrix by the maximum pixel value, to obtain a third matrix. Then, apixel value of each pixel point in the third matrix is multiplied by asecond pixel value of a corresponding pixel point in the second matrix,to obtain a fourth matrix. Then, the first pixel value of each pixelpoint in the first matrix is subtracted from the maximum pixel valuerespectively, to obtain a fifth matrix. Finally, the fourth matrix andthe fifth matrix are added, to obtain a sixth matrix. Wherein, a pixelvalue of each pixel point in the sixth matrix is the target pixel valueof the pixel point. As shown in formulas (2) and (3).

$\begin{matrix}{C = \frac{A_{mask}}{\max}} & (2) \\{D_{pixel} = {D + \left( {{MAX} - A_{mask}} \right)}} & (3)\end{matrix}$

wherein, C is the third matrix, A_(mask) is the first matrix of the maskimage, max is the maximum pixel value 255. D is the fourth matrix, isobtained by multiplying the pixel value of each pixel point in the thirdmatrix C by the second pixel value of the corresponding pixel point inthe second matrix B_(org); a value of each element in a matrix MAX isthe maximum pixel value, and the number of rows and the number ofcolumns of the matrix are the same as the number of rows and the numberof columns of the first matrix A_(mask); the fifth matrix is obtained byMAX−A_(mask); D_(pixel) denotes the sixth matrix.

It is noted that, in the above mentioned embodiment, the number of rowsand the number of columns of the fourth matrix are the same as thenumber of rows and the number of columns of the first matrix.

By the matrixes formed by the pixel values of the mask image and theoriginal image, the target pixel values of the pixel points arecomputed, which improves a computation speed.

S509, forming a target image of the pathologic change area according tothe target pixel values of all the pixel points.

After computing and obtaining the sixth matrix according to step S508,the target pixel value of each pixel point in the sixth matrix is put ina position corresponding to the pixel point in the original image or themask image, the target image of the pathologic change area may beformed.

S510, diagnosing the pathologic change area in the target image toacquire a diagnosis result.

The diagnosis process of S510 may be performed by a doctor, and it isalso possible to output the diagnosis result by Computer Aided Diagnosissoftware.

In some embodiments, it is possible to acquire an image of a pathologicchange area which has already been diagnosed as a sample image, andbased on a diagnosis result of the image, then use the sample image andits corresponding diagnosis result to construct a neural network fordiagnosis, so that the neural network for diagnosis converges or anerror is stabilized within an allowable error range. After obtaining awell trained neural network, it may be used to diagnose the patient.

Although, in the above description, the neural network is used toexplain implementation of the computer aided diagnosis, other machinelearning technologies may also be trained to form diagnosis models.

After obtaining the target image of the pathologic change area, thetarget image is input to a diagnosis model which is trained in advance,the diagnosis model diagnoses the pathologic change area in the targetimage, and outputs the diagnosis result.

In some embodiments, the image of the pathologic change area is input,as the image for diagnosis, to the diagnosis model for diagnosis, whichimproves the accuracy of the diagnosis result.

In some embodiments, in order to improve the robustness of the neuralnetwork for diagnosis, the method further includes a step of performingpretreatment on the sample image in training.

In some embodiments, a selected proportion of the sample images arerandomly selected to be subjected to a hair supplement process. Forexample, it is possible to select one fourth of the sample images to besubjected to the hair supplement process. For example, it is possible touse an image processing method to simulate drawing of hair, andsupplement that to skin areas in the sample images randomly according toa certain probability.

In some embodiments, a selected proportion of the sample images arerandomly selected to be subjected to a color enhancement process.Wherein, the color enhancement includes aspects of the color such assaturation, brightness and contrast, etc.

By performing the hair supplement process or the color enhancementprocess or both on a preset proportion of the sample images to increasethe diversity of the sample images, the accuracy of the output result ofthe trained neural network is improved.

In the image processing method provided by at least one embodiment ofthe present disclosure, position information of the pathologic changearea in the photographed image is obtained through the neural network,and the boundary of the pathologic change area is determined (in someembodiments, e.g., based on the obtained position information and theimage edge detection algorithm), the original image and the mask imagewhich include the pathologic change area are obtained, the originalimage and the mask image are fused, i.e., it is possible to obtain theimage which includes only the pathologic change area from the originalimage.

In the image processing method provided by at least one embodiment ofthe present disclosure, the first image for diagnosis is formed byscanning the skin part of the patient to be diagnosed which needsdiagnosis, the first image is input to the neural network to acquireposition information of the pathologic change area in the first image,and e.g., according to the position information and the image edgedetection algorithm, the boundary of the pathologic change area isdetermined, the original image and the mask image which include thepathologic change area are acquired from the first image, the mask imageand the original image are fused, to obtain the target imagecorresponding to the pathologic change area, wherein the target image isan image for diagnosing the pathologic change area, pixel points in thetarget image correspond to those in the original image and the maskimage one by one. By separating the pathologic change area from thenon-pathologic change area in the original image, the image whichincludes only the pathologic change area is obtained, so that it ispossible to position the pathologic change area accurately in the skinimage, and extract the image of the pathologic change area.

Some embodiments of the present disclosure also proposes an imageprocessing device.

As shown in FIG. 7, the image processing device includes: a firstacquisition module 710, a machine learning module 720, an extractionmodule 730, a fuse module 740.

The first acquisition module 710 is used to acquire a first image to bediagnosed, the first image including an image formed with respect to askin part of a patient to be diagnosed which needs diagnosis;

In some embodiments, the first acquisition module is integrated with askin imaging device, scans the skin part of the patient to be diagnosedwhich needs diagnosis, forms the first image for diagnosis.

In some embodiments, the skin imaging device scans the skin part of thepatient to be diagnosed which needs diagnosis, forms the first image fordiagnosis, the first acquisition module is coupled to the skin imagingdevice to acquire the formed first image.

The machine learning module 720 is used to input the first image into aneural network, to acquire position information of a pathologic changearea in the first image.

The extraction module 730 is used to acquire a boundary of thepathologic change area in the first image, acquire an original image anda mask image which include the pathologic change area from the firstimage.

The fuse module 740 is used to fuse the mask image and the originalimage, obtain a target image corresponding to the pathologic changearea, wherein the target image is an image for diagnosing the pathologicchange area, pixel points in the target image correspond to those in theoriginal image and the mask image one by one.

In some embodiments, as shown in FIG. 8, the fuse module 740 includes:an acquisition unit 741, a fuse unit 742, a formation unit 743.

The acquisition unit 741 is used to acquire a first pixel value of eachpixel point in the mask image and a second pixel value of each pixelpoint in the original image.

The fuse unit 742 is used to fuse the first pixel value of the pixelpoint in the mask image and the second pixel value of the correspondingpixel point in the original image, to acquire a target pixel value ofthe pixel point.

The formation unit 743 is used to form a target image of the pathologicchange area according to the target pixel values of all pixel points.

In some embodiments, the fuse unit 742 is also used to:

with respect to each pixel point, divide the first pixel value by amaximum pixel value to obtain a ratio, and multiply the ratio by thesecond pixel value, to obtain a first data;

acquire a difference value between the maximum pixel value and the firstpixel value;

add the first data and the difference value to obtain the target pixelvalue of the pixel point.

In some embodiments, the acquisition unit 741 is also used to:

extract a first pixel value of each pixel point in the mask image, anduse the first pixel value of each pixel point to constitute a firstmatrix of the mask image; wherein a position of the first pixel value ofeach pixel point in the first matrix is determined by a position of thepixel point in the mask image;

extract a second pixel value of each pixel point in the original image,and use the second pixel value of each pixel point to constitute asecond matrix of the original image; wherein a position of the secondpixel value of each pixel point in the second matrix is determined by aposition of the pixel point in the original image;

the fuse unit 742 is also used to:

divide the first matrix by the maximum pixel value, to obtain a thirdmatrix;

multiply a pixel value of each pixel point in the third matrix by thesecond pixel value of a corresponding pixel point in the second matrix,to obtain a fourth matrix;

subtract the first pixel value of each pixel point in the first matrixfrom the maximum pixel value respectively, to obtain a fifth matrix;

add the fourth matrix and the fifth matrix, to obtain a sixth matrix,wherein a pixel value of each pixel point in the sixth matrix is thetarget pixel value of the pixel point.

In some embodiments, the image processing device further includes:

a diagnosis module for inputting the target image to a diagnosis modelfor learning, diagnosing the pathologic change area in the target imageto acquire a diagnosis result.

In some embodiments, the image processing device further includes:

a second acquisition module for acquiring an image of a patientdiagnosed in the past as a sample image for training a constructedinitial neural network;

a third acquisition module for acquiring mark data of the sample image,the mark data including a position of a pathologic change area in thesample image;

a training module for inputting the sample images and the mark data intothe neural network for training, to form the neural network which has afunction of acquiring position information of the pathologic change areain the first image.

In some embodiments, the image processing device further includes:

a pretreatment module for randomly selecting a preset proportion of thesample images to be subjected to a hair supplement process; and/or,

randomly selecting a preset proportion of the sample images to besubjected to a color enhancement process.

It is to be noted that, the aforementioned explanation of theembodiments of the image processing method is also applicable to theimage processing device of the present embodiment, repeated descriptionis no longer made herein.

In the image processing device provided by at least one embodiment ofthe present disclosure, the first image for diagnosis is formed byscanning the skin part of the patient to be diagnosed which needsdiagnosis, the first image is input to the neural network for learningto acquire position information of the pathologic change area in thefirst image, the original image and the mask image which include thepathologic change area are acquired from the first image, the mask imageand the original image are fused, to obtain the target imagecorresponding to the pathologic change area, wherein the target image isan image for diagnosing the pathologic change area, pixel points in thetarget image correspond to those in the original image and the maskimage one by one. By fusing the original image and the mask image, it ispossible to separate the pathologic change area from the non-pathologicchange area in the original image, obtain the image which includes onlythe pathologic change area, so that it is possible to position andextract the pathologic change area accurately in the photographed image,reduce the labor cost.

Some embodiments of the present disclosure also proposes a computerdevice comprising: a processor; a memory; and computer programinstructions stored in the memory, which, when executed by theprocessor, cause the processor to execute one or more steps of the imageprocessing method provided by at least one embodiment of the presentdisclosure.

Some embodiments of the present disclosure also proposes a non-transientcomputer-readable storage medium with computer program instructionsstored thereon, which, when executed by a processor, cause the processorto execute one or more steps of the image processing method provided byat least one embodiment of the present disclosure.

Hereinafter, with reference to FIG. 9, a specific implementationstructure suitable to implement a computer device 800 of an embodimentof the present disclosure is illustrated. The computer device shown inFIG. 9 is merely an example, and does not limit a function and a userange of an embodiment of the present disclosure in any way.

As shown in FIG. 9, the computer system 800 includes one or moreprocessors 801, which may execute various operations according toprogram instructions stored in a memory 802 (e.g., program instructionsare stored in the memory 802 such as a read-only memory or a magneticdisk, etc, and are loaded into a random access memory (RAM)). In thememory 802, various programs and data required for operations of thecomputer system 800 are also stored. The processor 801, the memory 802are connected with each other through a bus 803. An input/output (I/O)interface 804 is also connected to the bus 803.

Various components may be connected to the I/O interface 804 to achieveoutput and output of information. E.g. an input device 805 whichincludes a keyboard, a mouse, etc; an output device 806 which includessuch as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), etc,as well as a speaker, etc; a communication device 807 which includes anetwork interface card such as a LAN card, a modem, etc. Thecommunication device 807 executes a communication process by way of anetwork such as Internet. A drive 808 is also connected to the I/Ointerface 803 according to needs. A removable medium 809, such as amagnetic disk, an optical disk, a Flash memory, etc, is connected to orinstalled in the drive 808 according to needs.

Wherein, the processor 801 may be a logic operation device which has adata process capability and/or a program execution capability, such as acentral processor (CPU) or a field programmable gate array (FPGA) or asingle-chip microcomputer (MCU) or a digital signal processor (DSP) oran application specific integrated circuit (ASIC), etc.

Wherein, the bus 803 may be Front Side Bus (FSB), QuickPath Interconnect(QPI), direct media interface (DMI), Peripheral Component Interconnect(PCI), Peripheral Component Interconnect Express (PCI-E), HyperTransport(HT), etc.

According to an embodiment of the present disclosure, the aboveprocesses described with reference to flow charts may be implemented ascomputer software programs. For example, an embodiment of the presentdisclosure includes a computer program product, the computer programproduct includes a computer program carried on a computer-readablemedium, the computer program includes program codes for executing theimage processing method of at least one embodiment of the presentdisclosure. In such an embodiment, the computer program may bedownloaded and installed through the communication device 807 from anetwork, and/or installed from the removable medium 809. When thecomputer program is executed by the processor 801, it executes the abovementioned functions defined in the system of the present disclosure.

In the description of the specification, the description with referenceto terms “one embodiment”, “some embodiments”, “an example”, “a specificexample”, or “some examples”, etc, means that specific features,structures, materials or characteristics described in conjunction withthe embodiment or the example are included in at least one embodiment orexample of the present disclosure. In the specification, exemplaryexpressions of the above mentioned terms do not necessarily refer to thesame embodiments or examples. Moreover, the described specific features,structures, materials or characteristics may be combined in a suitableway in any one or more embodiments or examples. In addition, in a casewhere there is no mutual contradiction, those skilled in the art mayintegrate and combine different embodiments or examples as well asfeatures of different embodiments or examples described in thespecification.

In addition, Terms “first”, “second” are only for the purpose ofdescription, can not be understood to indicate or imply relativeimportance or implicitly indicate the number of indicated technicalfeatures. Thus, features defined with “first”, “second” may explicitlyor implicitly include at least one of the features. In the descriptionof the present disclosure, “multiple” refers to at least two, e.g. two,three, etc, unless specifically limited otherwise.

Any process or method description described in a flow chart or herein inother ways may be understood to denote a module, segment or portion ofcode, which includes one or more executable instructions of steps forachieving customized logic functions or processes, and a range ofpreferable embodiments of the present disclosure includes additionalimplementations, wherein functions may be executed not in theillustrated or discussed order, including substantially concurrently orin the reverse order depending upon the functionality involved, thisshould be understood by those skilled in the art to which embodiments ofthe present disclosure belong.

Logic and/or steps shown in a flow chart or described herein in otherways, for example, may be considered to be a sequencing list ofexecutable instructions for implementing logical functions, may bespecifically embodied in any computer-readable medium, for use by, or inconnection with, an instruction execution system, device, or device(e.g., a system based on a computer, a system including a processor or asystem which fetches instructions from the instruction execution system,device, or apparatus and executes the instructions). In thespecification, a “computer-readable medium” may be any device that cancontain, store, communicate, propagate, or transport the program for useby or in connection with the instruction execution system, device, orapparatus. More specific examples (a non-exhaustive list) of thecomputer readable medium would include the following: an electricalconnection having one or more wires (an electronic device), a portablecomputer diskette (a magnetic device), a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor a Flash memory), an optical fiber device, and a portable compact discread-only memory (CD-ROM). In addition, the computer-readable medium mayeven be paper or another suitable medium upon which the program may beprinted, as the program can be electronically captured, via, forinstance, optical scanning of the paper or other medium, then compiled,interpreted, or otherwise processed in a suitable manner, if necessary,and then stored in a computer memory.

It should be understood that, respective parts of the present disclosuremay be embodied as hardware, software, firmware or a combinationthereof. In the above mentioned embodiments, multiple steps or methodsmay be embodied as software or firmware which is stored in the memoryand executed by a suitable instruction execution system. For example, ifthey are embodied as hardware in another embodiment, any of thefollowing technologies well-known in the art or a combination thereofmay be used: a discrete logic circuit which has a logic gate circuit forachieving a logical function with respect to data signals, anapplication specific integrated circuit which has suitable combinedlogic gate circuits, a programmable gate array (PGA), a fieldprogrammable gate array (FPGA), etc.

Those ordinary skilled in the art may understood that, all or a part of,steps carried in the above mentioned embodiment method may be executedby a program instructing relevant hardware, the program may be stored ina computer-readable storage medium, and when the program is executed, itincludes one of steps of the method embodiment or a combination thereof.

In addition, respective functional units in respective embodiments ofthe present disclosure may be integrated in one processing module,respective units may also exist independent physically, two or more thantwo of the above units may also be integrated in one module. The abovementioned integrated module may be embodied as hardware, may also beembodied as a software function module. When the integrated module isembodied as a software function module and is sold or used as anindependent product, it may also be stored in one computer-readablestorage medium.

The above mentioned storage medium may be a read-only memory, a magneticdisk or an optical disk, etc. While embodiments of the presentdisclosure have been illustrated and described above, it may beunderstood that, the above mentioned embodiments are exemplary, can notbe understood to limit the present disclosure, those ordinary skilled inthe art may make changes, alterations, substitutions and modificationsto the above mentioned embodiments within the scope of the presentdisclosure.

1. An image processing method comprising: acquiring a first image, thefirst image including an image formed with respect to a skin part of apatient to be diagnosed which needs diagnosis; inputting the first imageinto a neural network, to acquire position information of a pathologicchange area in the first image; acquiring a boundary of the pathologicchange area in the first image, acquiring an original image and a maskimage which include the pathologic change area from the first image; andfusing the mask image and the original image to obtain a target imagecorresponding to the pathologic change area; wherein the target image isan image for diagnosing the pathologic change area, and wherein pixelpoints in the target image have a one-to-one correspondence to pixelpoints in the original image and to pixel points in the mask image. 2.The image processing method according to claim 1, wherein fusing themask image and the original image to obtain the target imagecorresponding to the pathologic change area includes: acquiring a firstpixel value of each pixel point in the mask image and a second pixelvalue of each pixel point in the original image; fusing the first pixelvalue of each pixel point in the mask image and the second pixel valueof a corresponding pixel point in the original image, to acquire atarget pixel value of each pixel point; and forming the target image ofthe pathologic change area according to target pixel values of all pixelpoints.
 3. The image processing method according to claim 2, whereinfusing the first pixel value of each pixel point in the mask image andthe second pixel value of the corresponding pixel point in the originalimage to acquire a target pixel value of each pixel point includes:dividing the first pixel value of each pixel point in the mask image bya maximum pixel value to obtain a ratio, and multiplying the ratio bythe second pixel value of the corresponding pixel point in the originalimage, to obtain a first data; acquiring a difference value between themaximum pixel value and the first pixel value; and adding the first dataand the difference value to obtain the target pixel value of each pixelpoint.
 4. The image processing method according to claim 2, whereinacquiring the first pixel value of each pixel point in the mask imageand the second pixel value of each pixel point in the original imageincludes: extracting the first pixel value of each pixel point in themask image, and using the first pixel value of each pixel point toconstitute a first matrix of the mask image, wherein a position of thefirst pixel value of each pixel point in the first matrix is determinedby a position of the pixel point in the mask image, and extracting thesecond pixel value of each pixel point in the original image, and usingthe second pixel value of each pixel point to constitute a second matrixof the original image, wherein a position of the second pixel value ofeach pixel point in the second matrix is determined by a position of thepixel point in the original image; wherein fusing the first pixel valueof each pixel point in the mask image and the second pixel value of thecorresponding pixel point in the original image to acquire the targetpixel value of each pixel point includes: dividing the first matrix bythe maximum pixel value, to obtain a third matrix, multiplying a pixelvalue of each pixel point in the third matrix by the second pixel valueof a corresponding pixel point in the second matrix, to obtain a fourthmatrix, subtracting the first pixel value of each pixel point in thefirst matrix from the maximum pixel value respectively, to obtain afifth matrix, and adding the fourth matrix and the fifth matrix, toobtain a sixth matrix, wherein a pixel value of each pixel point in thesixth matrix is the target pixel value of each pixel point.
 5. The imageprocessing method according to claim 1, wherein after fusing the maskimage and the original image to obtain the target image corresponding tothe pathologic change area, the method further includes: diagnosing thepathologic change area in the target image to acquire a diagnosisresult.
 6. The image processing method according to claim 1, whereinbefore acquiring the first image, the method further includes: acquiringa sample image; acquiring mark data of the sample image, the mark dataincluding position information of the pathologic change area in thesample image; and training a neural network, by the sample image and themark data, to form the neural network which has a required function. 7.The image processing method according to claim 6, wherein beforetraining the neural network, by the sample image and the mark data, toform the neural network which has a required function, the methodfurther includes at least one of: randomly selecting a selectedproportion of the sample images to be subjected to a hair supplementprocess; or, randomly selecting a selected proportion of the sampleimages to be subjected to a color enhancement process.
 8. The imageprocessing method according to claim 1, wherein the position informationincludes center coordinates and a radius value of the pathologic changearea.
 9. The image processing method according to claim 1, wherein theposition information includes center coordinates, a major axis radiusvalue and a minor axis radius value of the pathologic change area. 10.The image processing method according to claim 1, before inputting thefirst image into the neural network to acquire the position informationof the pathologic change area in the first image, the method furtherincludes performing pretreatment on the first image.
 11. The imageprocessing method according to claim 10, wherein performing pretreatmenton the first image includes: acquiring a size of the first image;comparing the size of the first image with an image resolution parameterof an input layer of the neural network to determine whether the size ofthe first image is more than the image resolution parameter of the inputlayer; in response to determining that the size of the first image ismore than the image resolution parameter of the input layer, cutting orreducing the first image; and in response to determining that the sizeof the first image is less than the image resolution parameter of theinput layer, enlarging the first image.
 12. The image processing methodaccording to claim 1, wherein the step of acquiring the boundary of thepathologic change area in the first image to acquire the original imageand the mask image which include the pathologic change area from thefirst image is based on the position information and an image edgedetection algorithm.
 13. The image processing method according to claim1, wherein acquiring the first image includes scanning the skin part ofthe patient to be diagnosed which needs diagnosis to form the firstimage.
 14. The image processing method according to claim 1, whereinfusing the mask image and the original image includes performing abitwise AND operation on the mask image and the original image. 15.(canceled)
 16. A computer device comprising: a processor; a memory; andcomputer program instructions stored in the memory, which, when executedby the processor, cause the processor to execute the image processingmethod according to claim
 1. 17. A non-transient computer-readablestorage medium with computer program instructions stored thereon, which,when executed by a processor, cause the processor to execute the imageprocessing method according to claim
 1. 18. The computer device of claim16, wherein the fusing the mask image and the original image to obtainthe target image corresponding to the pathologic change area includes:acquiring a first pixel value of each pixel point in the mask image anda second pixel value of each pixel point in the original image; fusingthe first pixel value of each pixel point in the mask image and thesecond pixel value of a corresponding pixel point in the original image,to acquire a target pixel value of each pixel point; and forming thetarget image of the pathologic change area according to target pixelvalues of all pixel points.
 19. The computer device of claim 18, whereinthe fusing the first pixel value of each pixel point in the mask imageand the second pixel value of the corresponding pixel point in theoriginal image to acquire a target pixel value of each pixel pointincludes: dividing the first pixel value of each pixel point in the maskimage by a maximum pixel value to obtain a ratio, and multiplying theratio by the second pixel value of the corresponding pixel point in theoriginal image, to obtain a first data; acquiring a difference valuebetween the maximum pixel value and the first pixel value; and addingthe first data and the difference value to obtain the target pixel valueof each pixel point.
 20. The computer device of claim 18, whereinacquiring the first pixel value of each pixel point in the mask imageand the second pixel value of each pixel point in the original imageincludes: extracting the first pixel value of each pixel point in themask image, and using the first pixel value of each pixel point toconstitute a first matrix of the mask image, wherein a position of thefirst pixel value of each pixel point in the first matrix is determinedby a position of the pixel point in the mask image, and extracting thesecond pixel value of each pixel point in the original image, and usingthe second pixel value of each pixel point to constitute a second matrixof the original image, wherein a position of the second pixel value ofeach pixel point in the second matrix is determined by a position of thepixel point in the original image; wherein fusing the first pixel valueof each pixel point in the mask image and the second pixel value of thecorresponding pixel point in the original image to acquire the targetpixel value of each pixel point includes: dividing the first matrix bythe maximum pixel value, to obtain a third matrix, multiplying a pixelvalue of each pixel point in the third matrix by the second pixel valueof a corresponding pixel point in the second matrix, to obtain a fourthmatrix, subtracting the first pixel value of each pixel point in thefirst matrix from the maximum pixel value respectively, to obtain afifth matrix, and adding the fourth matrix and the fifth matrix, toobtain a sixth matrix, wherein a pixel value of each pixel point in thesixth matrix is the target pixel value of each pixel point.
 21. Thecomputer device of claim 16, herein after fusing the mask image and theoriginal image to obtain the target image corresponding to thepathologic change area, the method further includes: diagnosing thepathologic change area in the target image to acquire a diagnosisresult.